Skip to content

fix(eth-engine): mint a fresh JWT for each retried Engine API request - #394

Closed
Xowiek wants to merge 1 commit into
circlefin:mainfrom
Xowiek:fix/engine-jwt-per-attempt
Closed

Xowiek wants to merge 1 commit into
circlefin:mainfrom
Xowiek:fix/engine-jwt-per-attempt

Conversation

@Xowiek

@Xowiek Xowiek commented Sep 12, 2026

Copy link
Copy Markdown

EngineRpc::rpc_request mints the JWT once, before the request is sent:

.retry(retry_policy)
.bearer_auth(self.auth.generate_token()?)
.send()

RpcRequestBuilder::send then reuses that one string on every retry attempt, and exchange_capabilities retries with ENGINE_EXCHANGE_CAPABILITIES_RETRY_RPC — a constant 3s delay and without_max_times(), so it never gives up.

The Engine API rejects a token whose iat is more than 60 seconds from the server's clock (JWT_MAX_IAT_DIFF in alloy-rpc-types-engine, which is what reth's JwtAuthValidator checks). So if the execution client's auth port takes longer than a minute to come up, the consensus node keeps resending a token the EL can no longer accept, and no amount of further retrying recovers it — the node stays locked out even after the EL is healthy, until it is restarted.

Two attempts 1.1 seconds apart on main carry the byte-identical token:

left:  eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE3ODkyMzc4ODksImV4cCI6bnVsbH0...
right: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE3ODkyMzc4ODksImV4cCI6bnVsbH0...

The payload is {"iat":1789237889,"exp":null} in both — the timestamp is frozen at the first attempt.

The builder now takes the Auth instead of a finished token and mints one inside the retry closure. Only EngineRpc sets it; the IPC transport carries no JWT, and every other Engine method uses NoRetry, so nothing else changes.

The added test asserts the two attempts carry different tokens and that both validate against the secret. It fails on main with the output above. cargo test -p arc-eth-engine passes (96), as do fmt and clippy.

EngineRpc::rpc_request mints the token once, before the request is sent, and
RpcRequestBuilder::send reuses that one string on every retry attempt.
exchange_capabilities retries with a constant 3s delay and without_max_times(),
so it never gives up.

The Engine API rejects a token whose iat is more than 60 seconds from the
server's clock. If the execution client's auth port takes longer than that to
come up, the node keeps resending a token that can no longer be accepted, and
no further retry fixes it — even once the EL is healthy.

Take the Auth in the builder instead of a finished token and mint one per
attempt. Only EngineRpc sets it: the IPC transport carries no JWT and every
other Engine method uses NoRetry.
@ZhiyuCircle

Copy link
Copy Markdown
Contributor

Hi @Xowiek,

Thank you for your interest in contributing to Arc Node, and apologies for the delay in getting back to this PR.

We're closing out the pull request backlog that predates our current contribution policy. This PR is being closed because it does not reference a GitHub issue. All PRs must reference an existing issue using the format Closes: #XXX, and the author must be assigned to that issue before the PR is opened.

This is not a judgement on the change itself. If you'd still like to land it:

  1. Open an issue describing the problem, or find the existing one
  2. Comment on the issue requesting assignment, and wait for maintainer approval
  3. Open a fresh PR once you have been assigned

Please see CONTRIBUTING.md for details. Thanks again for taking the time to contribute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants